Temporal motion data candidate derivation in video coding

ABSTRACT

A method for derivation of a temporal motion data (TMD) candidate for a prediction unit (PU) in video encoding or video decoding is provided. The derived TMD candidate is for inclusion in an inter-prediction candidate list for the PU. The method includes determining a primary TMD position relative to a co-located PU in a co-located largest coding unit (LCU), wherein the co-located PU is a block in a reference picture having a same size, shape, and coordinates as the PU, and selecting at least some motion data of a secondary TMD position as the TMD candidate when the primary TMD position is in a bottom neighboring LCU or in a bottom right neighboring LCU of the co-located LCU, wherein the secondary TMD position is determined relative to the co-located PU.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/350,412, filed on Jun. 17, 2021, which is a continuation of U.S.patent application Ser. No. 15/460,037, filed on Mar. 15, 2017, which isa continuation of U.S. patent application Ser. No. 15/005,730, filed onJan. 25, 2016 (now U.S. Pat. No. 9,609,337), which is a continuation ofU.S. patent application Ser. No. 13/438,343, filed on Apr. 3, 2012 (nowU.S. Pat. No. 9,247,266), which claims the benefit of U.S. ProvisionalPatent Application No. 61/476,407, filed Apr. 18, 2011, and U.S.Provisional Patent Application No. 61/540,241, filed Sep. 28, 2011, allof which are incorporated herein by reference in their entirety.

BACKGROUND OF THE INVENTION Field of the Invention

Embodiments of the present invention generally relate to temporal motiondata candidate derivation in video coding.

Description of the Related Art

Video compression, i.e., video coding, is an essential enabler fordigital video products as it enables the storage and transmission ofdigital video. In general, video compression techniques applyprediction, transformation, quantization, and entropy coding tosequential blocks of pixels in a video sequence to compress, i.e.,encode, the video sequence. Video decompression techniques generallyperform the inverse of these operations in reverse order to decompress,i.e., decode, a compressed video sequence.

The Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T WP3/16and ISO/IEC JTC 1/SC 29/WG 11 is currently developing thenext-generation video coding standard referred to as High EfficiencyVideo Coding (HEVC). HEVC is expected to provide around 50% improvementin coding efficiency over the current standard, H.264/AVC, as well aslarger resolutions and higher frame rates. Several coding efficiencyenhancement tools are proposed in HEVC, among them a merge mode designedto reduce coding overhead by allowing an inter-predicted prediction unit(PU) to inherit motion data, i.e., motion vectors, prediction direction,and reference picture indices, from a position selected from neighboringmotion data positions in the same picture and a temporal motion dataposition derived based on a co-located block of the same size as the PUin a reference picture, referred to as the co-located PU. A skip mode isalso included that can be seen as a coding unit (CU) level merge modewith all zero transform coefficients. Regular motion vector coding forinter-prediction of a PU also considers motion vectors of selectedneighboring motion data positions in the same picture and a temporalmotion data position derived based on a co-located PU for use as motionvector predictors for the PU.

The temporal motion data from the co-located PU plays an important rolein improving the motion compensation efficiency for merge and skipmodes, and in motion vector prediction accuracy for differential motionvector coding of regular inter-predicted PUs. To derive a temporalmotion data candidate for the current PU, temporal motion data in ornear the co-located PU are fetched and scaled according to the temporaldistances. A co-located LCU/CU/PU is defined as a rectangular area in areference picture with the same coordinates, size, and shape as thecurrent LCU/CU/PU in the current picture. While the use of temporalmotion data for merge mode, skip mode, and regular motion vectorprediction does reduce coding overhead, additional improvements aredesirable.

SUMMARY

Embodiments of the present invention relate to methods and apparatus fortemporal motion data candidate derivation in video coding. In oneaspect, a method for derivation of a temporal motion data (TMD)candidate for a prediction unit (PU) in video encoding or video decodingis provided in which the TMD candidate is derived for inclusion in aninter-prediction candidate list for the PU. The method includesdetermining a primary TMD position relative to a co-located PU in aco-located largest coding unit (LCU), wherein the co-located PU is ablock in a reference picture having a same size, shape, and coordinatesas the PU, and selecting at least some motion data of a secondary TMDposition as the TMD candidate when the primary TMD position is in abottom neighboring LCU or in a bottom right neighboring LCU of theco-located LCU, wherein the secondary TMD position is determinedrelative to the co-located PU.

In one aspect, a method for derivation of a temporal merging candidatefor a prediction unit (PU) in video encoding or video decoding isprovided in which the temporal merging candidate is derived forinclusion in a merging candidate list for the PU. The method includesdetermining a primary temporal motion data (TMD) position relative to aco-located PU in a co-located largest coding unit (LCU), wherein theco-located PU is a block in a reference picture having a same size,shape, and coordinates as the PU and the primary TMD position is abottom right neighboring position outside the co-located PU, andselecting motion data of a secondary TMD position as the temporalmerging candidate when the primary TMD position is in a bottomneighboring LCU or in a bottom right neighboring LCU of the co-locatedLCU, wherein the secondary TMD position is inside the co-located PU.

In one aspect, a method for derivation of a temporal motion vectorpredictor (MVP) candidate for a prediction unit (PU) in video encodingor video decoding is provided in which the temporal MVP candidate isderived for inclusion in an advanced MVP (AMVP) candidate list for thePU. The method includes determining a primary temporal motion data (TMD)position relative to a co-located PU in a co-located largest coding unit(LCU), wherein the co-located PU is a block in a reference picturehaving a same size, shape, and coordinates as the PU and the primary TMDposition is a bottom right neighboring position outside the co-locatedPU, and selecting motion vector data from motion data of a secondary TMDposition as the temporal MVP candidate when the primary TMD position isin a bottom neighboring LCU or in a bottom right neighboring LCU of theco-located LCU, wherein the secondary TMD position is inside theco-located PU.

BRIEF DESCRIPTION OF THE DRAWINGS

Particular embodiments will now be described, by way of example only,and with reference to the accompanying drawings:

FIG. 1 is an example of decomposition of a largest coding unit (LCU)into coding units (CUs) and prediction units (PUs);

FIG. 2 is a block diagram illustrating prior art spatial and temporalmotion data positions for constructing a merging candidate list;

FIG. 3 is a block diagram of illustrating prior art spatial and temporalmotion data positions for constructing an advanced motion vectorpredictor (AMVP) candidate list;

FIG. 4 is a block diagram illustrating spatial and temporal motion datapositions for constructing a merging candidate list and an AMVPcandidate list;

FIG. 5 is an example LCU decomposition showing example temporal motiondata positions for PUs of the LCU;

FIG. 6 is an example illustrating temporal motion data fetch arearelative to a co-located LCU;

FIG. 7 is a block diagram of an example digital system;

FIG. 8 is a block diagram of a video encoder;

FIG. 9 is a block diagram of a video decoder;

FIG. 10 is an example illustrating a temporal motion data fetch arearelative to a co-located LCU when temporal motion data positions areconstrained to an LCU row;

FIGS. 11, 14, 17, 19, 21, and 23 are flow diagrams of methods fortemporal motion data derivation in an encoder or decoder;

FIGS. 12, 13, 15, 18, 20, 22, and 24 are example LCU decompositionsshowing example temporal motion data positions for PUs;

FIG. 16 is an example illustrating a temporal motion data fetch arearelative to a co-located LCU when temporal motion data positions areconstrained to the co-located LCU; and

FIG. 25 is a block diagram of an illustrative digital system.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION

Specific embodiments of the invention will now be described in detailwith reference to the accompanying figures. Like elements in the variousfigures are denoted by like reference numerals for consistency.

As used herein, the term “picture” may refer to a frame or a field of aframe. A frame is a complete image captured during a known timeinterval. For convenience of description, embodiments of the inventionare described herein in reference to HEVC. One of ordinary skill in theart will understand that embodiments of the invention are not limited toHEVC. In HEVC, a largest coding unit (LCU) is the base unit used forblock-based coding. A picture is divided into non-overlapping LCUs. Thatis, an LCU plays a similar role in coding as the macroblock ofH.264/AVC, but it may be larger, e.g., 32×32, 64×64, etc. An LCU may bepartitioned into coding units (CU). A CU is a block of pixels within anLCU and the CUs within an LCU may be of different sizes. Thepartitioning is a recursive quadtree partitioning. The quadtree is splitaccording to various criteria until a leaf is reached, which is referredto as the coding node or coding unit. The maximum hierarchical depth ofthe quadtree is determined by the size of the LCU and the size of thesmallest CU (SCU) permitted.

The coding node is the root node of two trees, a prediction tree and atransform tree. A prediction tree specifies the position and size ofprediction units (PU) for a coding unit. A PU is the basic unit forcarrying the information related to the prediction processes such asinter and intra-prediction. In general, a PU is not restricted to asquare shape in order to facilitate partitioning that matches boundariesof real objects in a picture. A CU may be partitioned into one or morePUs. A transform tree specifies the position and size of transform units(TU) for a coding unit. A transform unit may not be larger than a codingunit. The sizes of the transforms units and prediction units for a CUare determined by the video encoder during prediction based onminimization of rate/distortion costs. FIG. 1 shows an example of an LCUof size 64×64 that is decomposed into CUs and PUs. In this example, theSCU size is 16×16. In HEVC, the SCU size may be as small as 8×8.

As used herein, a co-located PU or temporally co-located PU is arectangular or square area in a reference picture having the samecoordinates, size, and shape of a PU in a picture currently beingencoded or decoded, i.e., a PU for which a merging candidate list or anadvanced motion vector predictor (AMVP) candidate list is beingconstructed. As is well-known, PU partitioning may change from LCU toLCU, and from picture to picture. Thus, a co-located PU does notnecessarily correspond to an actual PU of the reference picture. Rather,depending on the size, the co-located PU may overlap one actual PU,multiple actual PUs, portions of several actual PUs, a portion of anactual PU, etc. in the reference picture.

As used herein, a co-located CU or temporally co-located CU is a squarearea in a reference picture having the same coordinates, size, and shapeof a CU in a picture currently being encoded or decoded, i.e., a CU forwhich a merging candidate list is being constructed. As is well-known,CU partitioning may change from LCU to LCU, and from picture to picture.Thus, a co-located CU does not necessarily correspond to an actual CU ofthe reference picture. Rather, depending on the size, the co-located CUmay overlap one actual CU, multiple actual CUs, portions of severalactual CUs, a portion of an actual CU, etc. in the reference picture.

As used herein, a co-located LCU or temporally co-located LCU is asquare area in a reference picture having the same coordinates, size,and shape of an LCU in a picture currently being encoded or decoded,i.e., an LCU containing the PU for which a merging candidate list or anadvanced motion vector predictor (AMVP) candidate list is beingconstructed or the CU for which a merging candidate list is beingconstructed. As is well-known, LCU partitioning may change from pictureto picture if the two pictures refer to different sequence parametersets (SPS). Thus, a co-located LCU does not necessarily correspond to anactual LCU of the reference picture. Rather, depending on the size, theco-located LCU may overlap one actual LCU, multiple actual LCUs,portions of several actual LCUs, a portion of an actual LCU, etc. in thereference picture.

Some aspects of this disclosure have been presented to the JCT-VC in thefollowing documents: M. Zhou, “Non-CE9: Modified H Positions for MemoryBandwidth Reduction in TMVP Derivation” JCTVC-G082, Nov. 19-30, 2011,and M. Zhou, “CE1: Evaluation Results on A.09, A.13-16 and anAlternative Solution”, JCTVC-F081, Jul. 14-22, 2011, which areincorporated by reference herein in their entirety.

As previously discussed, merge mode, skip mode, and regular motionvector coding based on spatially neighboring PUs and a temporallyco-located PU for inter-prediction of PUs are proposed in HEVC. Generaldescriptions of merge mode, skip mode, and regular motion vector codingare provided herein. More detailed descriptions of the emerging proposalmay be found in K. McCann, et al., “High Efficiency Video Coding (HEVC)Test Model 2 (HM 2) Encoder Description,” JCTVC-D502, JointCollaborative Team on Video Coding (JCT-VC) of ITU-T SG16 WP3 andISO/IEC JTC1/SC29/WG11, Guangzhou, CN, Oct. 7-15, 2010, and T. Wiegand,et al., “WD2: Working Draft 2 of High-Efficiency Video Coding,”JCTVC-D503, Joint Collaborative Team on Video Coding (JCT-VC) of ITU-TSG16 WP3 and ISO/IEC JTC1/SC29/WG11, Daegu, Korea, Jan. 20-28, 2011(“WD2”), T. Wiegand, et al., “WD3: Working Draft 3 of High-EfficiencyVideo Coding,” JCTVC-E603, Joint Collaborative Team on Video Coding(JCT-VC) of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11, Geneva, CH, Mar.16-23, 2011 (“WD3”), B. Bross, et al., “WD4: Working Draft 4 ofHigh-Efficiency Video Coding, JCTVC-F803_d6, Joint Collaborative Team onVideo Coding (JCT-VC) of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11,Torino, IT, Jul. 14-22, 2011 (“WD4”), B. Bross. et al., “WD5: WorkingDraft 5 of High-Efficiency Video Coding, JCTVC-G1103_d9, JointCollaborative Team on Video Coding (JCT-VC) of ITU-T SG16 WP3 andISO/IEC JTC1/SC29/WG11, Geneva, CH, Nov. 21-30, 2011 (“WD5”), and B.Bross, et al., “High Efficiency Video Coding (HEVC) Text SpecificationDraft 6, JCTVC-H1003, Joint Collaborative Team on Video Coding (JCT-VC)of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG1, Geneva, CH, Nov. 21-30,2011 (“HEVC Draft 6”), all of which are incorporated by referenceherein.

In HEVC, the minimum PU size is 4×4, i.e., samples in a 4×4 region sharea same set of motion data when the PU is inter-predicted, and aninter-predicted PU of a larger size can be treated as a multiple of 4×4blocks which share the motion data of the PU. For simplicity of theimplementation, for advanced motion vector predicator (AMVP) and mergingcandidate list derivation, the motion data may be populated to the 4×4block level regardless of PU size. For example, if a neighboringinter-predicted PU is 16×16, the same set of PU motion data isreplicated 16 times for the 4×4 blocks of the PU and stored. In thisway, the motion data from neighboring 4×4 blocks in the relevantlocations can be fetched to derive the AMVP or merging candidateswithout the need to know the actual neighboring PU sizes to compute themotion data location. In WD3, WD4, and WD5, the motion data is populatedto sample (pixel) level for the simplicity of description. In actualimplementation, such a motion data storage format, i.e., sample-based,may not be used due to memory usage considerations. In the descriptionherein, the various motion data blocks specify the locations from whichthe neighboring motion data, both spatial and temporal, should befetched. The encoder or decoder has the responsibility to determinewhich PU contains a motion data block and to fetch the PU motion dataaccordingly. The simplest way to realize this functionality, and toavoid an on-the-fly computation, is to store the motion data for a PU atthe 4×4 block level once motion data is determined for that PU.

In general, merge mode allows an inter-predicted PU to inherit the samemotion vector(s), prediction direction, and a reference picture index(or indices) from an inter-predicted PU which contains a motion dataposition selected from a group of spatially neighboring motion datapositions and one of two temporally co-located motion data positions.FIG. 2 illustrates candidate motion data positions for the merge mode asdefined in WD3. For the current PU 200, the encoder forms a mergingcandidate list by considering merging candidates from the motion datapositions depicted in FIG. 2: four spatially neighboring motion data(SMD) positions, i.e., a left neighboring SMD position 204, an upperneighboring SMD position 206, an upper right neighboring SMD position208, and a bottom left neighboring SMD position 210, and two temporalmotion data (TMD) positions of the a temporally co-located PU 202.

To choose the co-located temporal merging candidate, the co-locatedtemporal motion data from the bottom right TMD position (see 3 (BR) inFIG. 2, outside the co-located PU 202) is first checked and selected forthe temporal merging candidate if available. Otherwise, the co-locatedtemporal motion data at the upper left central TMD position (see 3 (CR)in FIG. 2) is checked and selected for the temporal merging candidate ifavailable. To derive the motion data for a merging candidate from amotion data position, the needed motion data is copied from thecorresponding PU which contains (or covers) the motion data position.The merging candidates in the list, if available, are ordered in themerging candidate list as numbered in FIG. 2, with the merging candidatefrom the left neighboring SMD position 204 placed at the beginning ofthe list, the temporal merging candidate from the TMD position bottomright to or inside the co-located PU 202, in the third position, andthat of the bottom left neighboring SMD position 210 placed at the endof the list. The derivation of the spatially neighboring mergingcandidates, the temporal neighboring merging candidate, and the criteriafor availability are explained in WD3.

A merging candidate includes motion vector information, prediction flaginformation, and reference picture index information for a candidatemotion data position. A merging candidate may include sufficient entriesto accommodate a bi-directionally predicted PU, i.e., entries for aforward motion vector, a backward motion vector, a forward referencepicture index, a backward reference picture index, and a prediction flagindicating prediction direction, i.e., forward, backward, orbi-directional. The prediction flag may be composed of two predictionlist utilization flags used to indicate which of two reference picturelists is to be used. Each reference picture index is an index into arespective one of the reference picture lists. For a motion dataposition contained by a forward predicted PU, the merging candidateentries for the prediction flag, the forward motion vector, and theforward reference picture index will be valid and the remaining entriesmay have placeholder values. For a motion data position contained by abackward predicted PU, the merging candidate entries for the predictionflag, the backward motion vector, and the backward reference pictureindex will be valid and the remaining entries may have placeholdervalues. For a bi-directionally predicted PU, all merging candidateentries will be valid.

In HEVC, the merging candidate entries may be referred to according totheir correspondence with one of two reference picture lists, list 0 andlist 1. Thus, the forward motion vector may be referred to as the list 0(or L0) motion vector, the backward motion vector may be referred to asthe list 1 (or L1) motion vector, the two prediction list utilizationflags be referred to as the list 0 (or L0) prediction list utilizationflag and the list 1 (or L1) prediction list utilization flag, and thereference picture indices may be referred to as the list 0 (or L0)reference picture index and the list 1 (or L1) reference picture index.

After the merging candidate list is formed, a pruning process is carriedout to remove any duplicated merging candidates. If two or more mergingcandidates have the same motion vector(s), prediction direction, andreference picture index (or indices), the lowest order duplicatedmerging candidate is retained in the list and the others are removed. Ifall the merging candidates are not valid, zero motion vector mergingcandidates are added to the merging candidate list. Therefore, themerging candidate list size for merge mode may be of size 1, 2, 3, 4 or5. Invalidity of a merging candidate for merge mode is explained in WD3.

In general, skip mode allows the encoder to “skip” coding of aninter-predicted CU when it can be effectively inter-predicted frommotion data of a neighboring PU or a temporally co-located CU. Morespecifically, skip mode allows an inter-predicted CU to inherit themotion data of a spatial or temporal neighbor, and no non-zero quantizedtransform coefficients are encoded for the CU. Skip mode is determinedat the CU level and is essentially a merge mode at the CU-level withoutnon-zero transform coefficients. Thus, for skip mode, the encodergenerates a merging candidate list as previously described except thatthe current PU is a CU. The same relative positions for the spatialmerging candidates and the temporal merging candidate are used. Amerging candidate for skip mode also contains the same information aspreviously described for a merging candidate.

In general, for direct or normal inter-prediction, motion vector(s) of aPU is (are) predicatively coded relative to a motion vector predictor(s)(MVP(s)) from an advanced motion vector predictor (AMVP) candidate listconstructed by the encoder. For single direction inter-prediction of aPU, the encoder generates a single AMVP candidate list. Forbi-directional prediction of a PU, the encoder generates two AMVPcandidate lists, one using motion data of spatial and temporalneighboring PUs from the forward prediction direction and one usingmotion data of spatial and temporal neighboring PUs from the backwardprediction direction.

FIG. 3 illustrates the formation of an AMVP candidate list for thecurrent PU 300 as defined in WD3. The encoder forms an AMVP candidatelist based on neighboring SMD positions and TMD positions of aco-located PU 302 as illustrated in the example of FIG. 3. The motionvectors for a motion data position are selected as an MVP from themotion data of the corresponding PU which contains (covers) the motiondata position. For the spatial MVP candidate derivation, the SMDpositions to the left of the current PU 300 are scanned bottom up, e.g.,from the bottom left SMD position 304 to the left top SMD position 306,and the motion vector of the first SMD position on the left side havingavailable motion data is chosen to be the first candidate MVP for theAMVP candidate list. Then, the upper side neighboring SMD positions arescanned left to right, e.g., from the top right SMD position 312,through the left top SMD position 310, ending with the top left SMDposition 308. The motion vector of the first SMD position on the upperneighboring side having available motion data with a motion vector of adifferent value from the first candidate MVP is chosen as the secondcandidate MVP in the AMVP candidate list. If no spatial MVP candidate isfound during the scan of the left-side SMD positions, then up to two MVPcandidates may be selected from the top-side SMD positions. That is, thefirst available motion vector of the upper left side is chosen as thefirst candidate MVP in the AMVP candidate list and the second availablemotion vector different from the first is chosen as the second candidateMVP in the AMVP candidate list.

To chose the temporal candidate MVP, the availability of motion datafrom the bottom right TMD position of the co-located PU 302 (see 3 (BR)in FIG. 3, outside the co-located PU 202) is first checked and themotion vector selected for the temporal candidate MVP if available.Otherwise, the availability of motion data at the upper left central TMDposition of the co-located PU 302 (see 3 (CR) in FIG. 3) is checked andthe motion vector selected for the temporal candidate MVP if available.Note that this is essentially the same derivation process as that usedto select the temporal merging candidate in the merging candidate listderivation process. The temporal MVP candidate is added to the AMVPcandidate list in the third position. The derivation of the spatial MVPcandidates, the temporal MVP candidate, and the criteria foravailability for the AMVP candidate list are explained in WD3.

If no candidate MVPs are found in the scans of the left/upper SMDpositions and from co-located temporal PU, a zero MVP is added to theAMVP candidate list. After the AMVP candidate list is formed, a pruningprocess similar to that used in pruning the merging candidate list iscarried out to remove any duplicated MVP candidates. Therefore, the AMVPcandidate list size may be 1, 2, or 3.

In WD4, WD5, and HEVC Draft 6, the derivation of the merging candidatelist and the AMVP candidate list was changed to use the same SMDpositions in constructing each list. The co-located PU and the relativeTMD positions remained the same. FIG. 4 shows the five SMD positions ofthe current PU 400 and the temporally co-located PU 402. For the mergingcandidate list, the merging candidates, if available, are ordered in thelist as per the numbering in FIG. 4. For the AMVP candidate list, theMVPs of the motion data positions, if available, are ordered in the listas per the numbering in FIG. 4. Pruning of the lists and the addition ofzero motion vector merging candidates or zero MVPs to the respectivelists is as previously described. The derivation of the spatialcandidates, the temporal candidate, and the criteria for availabilityfor each list are explained in WD4, WD5, and HEVC Draft 6.

In general, for a CU, the encoder generates a merging candidate list forskip mode, a merging candidate list for each PU in the CU, and one ortwo AMVP candidate lists for each PU in the CU. The encoder then usesthe best candidates in each list in the determination of rate/distortion(RD) costs for using each mode. For each PU, the encoder selects thebetter mode between merge and normal inter-predicted mode based on theRD costs. The sum of the costs for the selected modes for all PUs in theCU is the RD cost for the CU for inter-predicted mode, i.e., non-skippedand non-intra coded mode. At the CU level, the encoder chooses the bestmode among skip mode, inter-predicted mode, and Intra-predicted modebased on the RD costs of each.

For each inter-predicted CU, the encoder encodes a skip flag into thebit stream to signal whether or not the current CU is coded with skipmode. If skip mode is used, the encoder also encodes the index in themerging candidate list generated for skip mode of the merging candidateselected (unless there is only one entry in the list). If skip mode isnot used for the CU and intra-prediction is not selected, the encoderencodes a merge flag into the bit stream for each inter-predicted PU ofthe CU to signal whether or not the merge mode is used for the PU. Ifmerge mode is used, the encoder also encodes the index in the mergingcandidate list of the merging candidate selected for merging (unlessthere is only one entry in the list). If merge mode is not used, theencoder encodes the normal inter-prediction information for the PU inthe bit-stream such as an index (or indices) into the AMVP candidatelist(s) for the MVP candidate(s) selected for differential encoding ofthe motion vector(s), prediction direction(s), motion vector differences(MVDs), and the reference picture index (or indices).

The decoder is also required to construct a merging candidate list,and/or up to two AMVP candidate lists when decoding an inter-predictedPU, depending upon which mode was used for inter-prediction in theencoder, and a merging candidate list for an inter-predicted CU whenskip mode was used by the encoder. The construction of these lists isthe same as that performed in the encoder.

As has been explained, the derivation of the temporal merging candidateand the temporal MVP candidate is the same except for the amount ofmotion data used for a candidate, i.e., a temporal MVP candidate usesthe motion vector data from the PU covering a motion data position and atemporal merging candidate uses the motion vector data and additionaldata from the PU covering a motion data position. For simplicity ofexplanation, the generic term temporal motion data (TMD) candidate isused when the description applies to both of these candidates. Further,the term inter-prediction candidate list used when the descriptionapplies to both a merging candidate list and an AMVP candidate list.

In a practical implementation of an HEVC encoder or decoder, for thederivation of TMD candidates for the current LCU, the temporal motiondata (motion vectors, reference picture index, prediction direction,prediction size) needs to be pre-fetched for the entire LCU. Using thederivation of the TMD candidates as previously described, thepre-fetched co-located temporal motion data for the current LCU cancross LCU boundaries in the co-located reference picture. Consider theexample of FIG. 5, in which the current LCU is decomposed into four CUs,CU0-CU3, which are further divided into nine PUs of different sizes,PU0-PU8. Because the bottom right (BR) TMD position, which is outsidethe co-located PU, is used for derivation of the TMD candidate for thecurrent PU, the TMD positions 500, 502, 504, 506, 508 are outside theco-located LCU in the reference picture. Consequently, for thederivation of the TMD candidates for the current LCU, not only themotion data from the co-located LCU but also from its neighboring LCUs,i.e., the right, bottom, and bottom right neighboring LCUs needs to befetched. In an embedded system implementation of an encoder or decoder,the motion data for reference pictures is stored in off-chip memory.Thus, the need to fetch all of these reference LCUs consumes significantmemory bandwidth and cycle overhead.

Put another way, the fetch area for the temporal motion data needed forTMD candidate derivation is not aligned with the current LCU. Because ofthe bottom right (BR) TMD candidate position, this fetch area can gobeyond the boundaries of the current LCU. As shown in FIG. 6, an 80×80co-located motion data area 600 needs to be pre-fetched for a 64×64 LCU602. The 80×80 co-located area 602 touches motion data of four LCUs inthe co-located picture.

HEVC uses motion data compression to decrease memory bandwidth. Toensure memory burst alignments and avoid page crossing, thus minimizingmemory bandwidth requirements, compression, storage, and fetching ofmotion data in 64×64 LCU aligned tile format is desirable. As shown inTable 1, after compression, the motion data size for each 16×16 blockarea is about 74 bits which is approximately 10 bytes. For a 64×64 LCU,the total motion data after compression is about 148 bytes, which isapproximately 19, 10, and 5 bursts for a memory burst size of 8, 16, and32 bytes, respectively.

TABLE 1 Motion data Size of data after compression for 16 × 16 block(bits) Motion vectors 64 (2 × 2 × 16) Intra/inter flag + Prediction 2direction Ref_idx 8 (2 × 4) Total for 16 × 16 (bits) 74 (bits) = ~10(bytes) Total for 64 × 64 LCU (bytes) 148 (bytes)

If the motion data is compressed and stored in 64×64 LCU aligned tileformat, motion data from four LCUs in the co-located picture is fetchedfor the derivation of TMD candidates of the current LCU, thus incurringhigh memory bandwidth. One solution to lower the memory bandwidth is toadd a line buffer to maintain the co-located motion data for theprevious LCU row, so that the motion data can be re-used by the TMDcandidate derivation process of the current LCU row. However, such asolution is expensive.

Embodiments of the invention provide for derivation of TMD candidatesthat resolve the misalignment issue. More specifically, some embodimentsprovide an alternative TMD candidate derivation in which the TMDpositions are constrained to be within the boundaries of the co-locatedLCU row. Other embodiments provide an alternative TMD derivation inwhich the TMD positions are constrained to be within the boundaries ofthe co-located LCU.

FIG. 7 shows a block diagram of a digital system that includes a sourcedigital system 700 that transmits encoded video sequences to adestination digital system 702 via a communication channel 716. Thesource digital system 700 includes a video capture component 704, avideo encoder component 706, and a transmitter component 708. The videocapture component 704 is configured to provide a video sequence to beencoded by the video encoder component 706. The video capture component704 may be, for example, a video camera, a video archive, or a videofeed from a video content provider. In some embodiments, the videocapture component 704 may generate computer graphics as the videosequence, or a combination of live video, archived video, and/orcomputer-generated video.

The video encoder component 706 receives a video sequence from the videocapture component 704 and encodes it for transmission by the transmittercomponent 708. The video encoder component 706 receives the videosequence from the video capture component 704 as a sequence of pictures,divides the pictures into largest coding units (LCUs), and encodes thevideo data in the LCUs. The video encoder component 706 may beconfigured to perform temporal motion data (TMD) candidate derivationduring the encoding process as described herein. An example of the videoencoder component 706 is described in more detail herein in reference toFIG. 8.

The transmitter component 708 transmits the encoded video data to thedestination digital system 702 via the communication channel 716. Thecommunication channel 716 may be any communication medium, orcombination of communication media suitable for transmission of theencoded video sequence, such as, for example, wired or wirelesscommunication media, a local area network, or a wide area network.

The destination digital system 702 includes a receiver component 710, avideo decoder component 712 and a display component 714. The receivercomponent 710 receives the encoded video data from the source digitalsystem 700 via the communication channel 716 and provides the encodedvideo data to the video decoder component 712 for decoding. The videodecoder component 712 reverses the encoding process performed by thevideo encoder component 706 to reconstruct the LCUs of the videosequence. The video decoder component 712 may be configured to performTMD candidate derivation during the decoding process as describedherein. An example of the video decoder component 712 is described inmore detail below in reference to FIG. 9.

The reconstructed video sequence is displayed on the display component714. The display component 714 may be any suitable display device suchas, for example, a plasma display, a liquid crystal display (LCD), alight emitting diode (LED) display, etc.

In some embodiments, the source digital system 700 may also include areceiver component and a video decoder component and/or the destinationdigital system 702 may include a transmitter component and a videoencoder component for transmission of video sequences both directionsfor video steaming, video broadcasting, and video telephony. Further,the video encoder component 706 and the video decoder component 712 mayperform encoding and decoding in accordance with one or more videocompression standards. The video encoder component 706 and the videodecoder component 712 may be implemented in any suitable combination ofsoftware, firmware, and hardware, such as, for example, one or moredigital signal processors (DSPs), microprocessors, discrete logic,application specific integrated circuits (ASICs), field-programmablegate arrays (FPGAs), etc.

FIG. 8 shows a block diagram of the LCU processing portion of an examplevideo encoder. A coding control component (not shown) sequences thevarious operations of the LCU processing, i.e., the coding controlcomponent runs the main control loop for video encoding. The codingcontrol component receives a digital video sequence and performs anyprocessing on the input video sequence that is to be done at the picturelevel, such as determining the coding type (I, P, or B) of a picturebased on the high level coding structure, e.g., IPPP, IBBP,hierarchical-B, and dividing a picture into LCUs for further processing.The coding control component also may determine the initial LCU CUstructure for each CU and provides information regarding this initialLCU CU structure to the various components of the video encoder asneeded. The coding control component also may determine the initial PUand TU structure for each CU and provides information regarding thisinitial structure to the various components of the video encoder asneeded.

The LCU processing receives LCUs of the input video sequence from thecoding control component and encodes the LCUs under the control of thecoding control component to generate the compressed video stream. TheCUs in the CU structure of an LCU may be processed by the LCU processingin a depth-first Z-scan order. The LCUs 800 from the coding control unitare provided as one input of a motion estimation component 820, as oneinput of an intra-prediction component 824, and to a positive input of acombiner 802 (e.g., adder or subtractor or the like). Further, althoughnot specifically shown, the prediction mode of each picture as selectedby the coding control component is provided to a mode selector componentand the entropy encoder 834.

The storage component 818 provides reference data to the motionestimation component 820 and to the motion compensation component 822.The reference data may include one or more previously encoded anddecoded CUs, i.e., reconstructed CUs.

The motion estimation component 820 provides motion data information tothe motion compensation component 822 and the entropy encoder 834. Morespecifically, the motion estimation component 820 performs tests on CUsin an LCU based on multiple inter-prediction modes (e.g., skip mode,merge mode, and normal or direct inter-prediction) and transform blocksizes using reference picture data from storage 818 to choose the bestmotion vector(s)/prediction mode based on a rate distortion coding cost.To perform the tests, the motion estimation component 820 may begin withthe CU structure provided by the coding control component. The motionestimation component 820 may divide each CU indicated in the CUstructure into PUs according to the unit sizes of prediction modes andinto transform units according to the transform block sizes andcalculate the coding costs for each prediction mode and transform blocksize for each CU. The motion estimation component 820 may also computeCU structure for the LCU and PU/TU partitioning structure for a CU ofthe LCU by itself.

As mentioned above, the prediction modes considered by the motionestimation component 820 may be merge mode, skip mode, and regular(normal) inter-prediction mode. To consider skip mode, the motionestimation component 820 constructs a skip mode merging candidate listat the CU level. To consider merge mode, the motion estimation component820 constructs a merging candidate list for each PU in the CU. Toconsider regular inter-prediction mode, the motion estimation component820 estimates motion vectors and constructs one or two AMVP candidatelists (depending on prediction direction) for each PU. For simplicity ofexplanation, the SMD positions and the ordering of the mergingcandidates in the merging candidate list and the AMVP candidate list areassumed to be as shown in FIG. 4. The derivation of the spatial mergingcandidates, the spatial MVP candidates, and the criteria foravailability may be as described in WD4, WD5, or HEVC Draft 6. One ofordinary skill in the art will understand that different SMD positions,different derivations, and/or different ordering may be used. Thederivation of the temporal merging candidate for each candidate list maybe performed as per methods described herein.

For each PU of a CU, the motion estimation component 820 computes codingcosts for each entry in the merging candidate list and selects the entrywith the best result. The coding cost of this entry is used by themotion estimation component 820 in prediction mode selection. For eachPU of the CU, the motion estimation component 820 determines the bestmotion vectors and MVP(s) from the AMVP candidate list(s) based oncoding costs, and uses the best coding cost for prediction modeselection. For each PU in the CU, the motion estimation componentselects the better of merge mode and normal inter-predicted mode basedon the coding costs. The sum of the costs of the selected modes for allPUs in the CU is the RD cost for the CU in inter-predicted mode. For theCU, the motion estimation component 820 also computes coding costs foreach entry in the skip mode merging candidate list and selects the entrywith the best result. The coding cost of this entry is used by themotion estimation component 820 in prediction mode selection betweenCU-level skip mode and normal inter-predicted mode.

For coding efficiency, the motion estimation component 820 may alsodecide to alter the CU structure by further partitioning one or more ofthe CUs in the CU structure. That is, when choosing the best motionvectors/prediction modes, in addition to testing with the initial CUstructure, the motion estimation component 820 may also choose to dividethe larger CUs in the initial CU structure into smaller CUs (within thelimits of the recursive quadtree structure), and calculate coding costsat lower levels in the coding hierarchy. If the motion estimationcomponent 820 changes the initial CU structure, the modified CUstructure is communicated to other components that need the information.

The motion estimation component 820 provides the selected motion vector(MV) or vectors and the selected prediction mode for eachinter-predicted PU of a CU to the motion compensation component 822 andthe selected motion vector (MV), reference picture index (indices),prediction direction (if any) to the entropy encoder 834. If merge modeor skip mode provides the best motion vector(s)/prediction mode for a PUor CU based on a coding cost, the motion estimation component 820 alsoindicates to the entropy encoder 834 to encode a merge (skip) flagindicating that merge (skip) mode is used for a PU (CU) and to encode anindex into the merging candidate list for the entry that provided thebest coding cost. The index may not be encoded if the candidate listsize is one; instead it is inferred to be 0.

If merge mode did not provide the best coding cost for aninter-predicted PU, the motion estimation component 820 indicates to theentropy encoder 834 to encode a merge flag indicating that merge modewas not used for the PU. A merge flag is encoded for eachinter-predicted PU unless skip mode is selected for the CU containingthe PU. Further, if normal inter-prediction mode provided the bestcoding cost, the motion estimation component 820 indicates to theentropy encoder 834 to encode an index (or indices) into the AMVPcandidate list(s) for the MVP candidate(s) used for differentialprediction of the motion vector(s).

The motion compensation component 822 provides motion compensatedinter-prediction information to the mode decision component 826 thatincludes motion compensated inter-predicted PUs, the selectedinter-prediction modes for the inter-predicted PUs, and correspondingtransform block sizes. The coding costs of the inter-predicted PUs arealso provided to the mode decision component 826.

The intra-prediction component 824 provides intra-prediction informationto the mode decision component 826 that includes intra-predicted PUs andthe corresponding intra-prediction modes. That is, the intra-predictioncomponent 824 performs intra-prediction in which tests based on multipleintra-prediction modes and transform unit sizes are performed on CUs inan LCU using previously encoded neighboring PUs from the buffer 828 tochoose the best intra-prediction mode for each PU in the CU based on acoding cost. To perform the tests, the intra-prediction component 824may begin with the CU structure provided by the coding control. Theintra-prediction component 824 may divide each CU indicated in the CUstructure into PUs according to the unit sizes of the intra-predictionmodes and into transform units according to the transform block sizesand calculate the coding costs for each prediction mode and transformblock size for each PU.

For coding efficiency, the intra-prediction component 824 may alsodecide to alter the CU structure by further partitioning one or more ofthe CUs in the CU structure. That is, when choosing the best predictionmodes, in addition to testing with the initial CU structure, theintra-prediction component 824 may also chose to divide the larger CUsin the initial CU structure into smaller CUs (within the limits of therecursive quadtree structure), and calculate coding costs at lowerlevels in the coding hierarchy. If the intra-prediction component 824changes the initial CU structure, the modified CU structure iscommunicated to other components that need the information. Further, thecoding costs of the intra-predicted PUs and the associated transformblock sizes are also provided to the mode decision component 826.

The mode decision component 826 selects between the motion-compensatedinter-predicted PUs from the motion compensation component 822 and theintra-predicted PUs from the intra-prediction component 824 based on thecoding costs of the PUs and the picture prediction mode provided by themode selector component. The decision is made at CU level. Based on thedecision as to whether a CU is to be intra- or inter-coded, theintra-predicted PUs or inter-predicted PUs are selected, accordingly.The output of the mode decision component 826, i.e., the predicted PU,is provided to a negative input of the combiner 802 and to a delaycomponent 830. The associated transform block size is also provided tothe transform component 804. The output of the delay component 830 isprovided to another combiner (i.e., an adder) 838. The combiner 802subtracts the predicted PU from the current PU to provide a residual PUto the transform component 804. The resulting residual PU is a set ofpixel difference values that quantify differences between pixel valuesof the original PU and the predicted PU. The residual blocks of all thePUs of a CU form a residual CU block for the transform component 804.

The transform component 804 performs block transforms on the residual CUto convert the residual pixel values to transform coefficients andprovides the transform coefficients to a quantize component 806. Thetransform component 804 receives the transform block sizes for theresidual CU and applies transforms of the specified sizes to the CU togenerate transform coefficients.

The quantize component 806 quantizes the transform coefficients based onquantization parameters (QPs) and quantization matrices provided by thecoding control component and the transform sizes. The quantizedtransform coefficients are taken out of their scan ordering by a scancomponent 808 and arranged by significance, such as, for example,beginning with the more significant coefficients followed by the lesssignificant.

The ordered quantized transform coefficients for a CU provided via thescan component 808 along with header information for the CU are coded bythe entropy encoder 834, which provides a compressed bit stream to avideo buffer 836 for transmission or storage. The header information mayinclude the prediction mode used for the CU. If the CU uses merge mode,and all the transform coefficients after quantization are all zero, theCU is coded with skip mode, a skip flag equal to one is encoded into bitstream, and an index into the merging candidate list for the mergingcandidate used for the skip mode is also encoded unless the size of themerging candidate list is one. Otherwise, a merge flag is encoded foreach PU of the CU unless the CU is intra-coded. Further, if merge modeis actual mode selected for prediction of a PU, an index into themerging candidate list for the merging candidate used for prediction ofthe PU is also encoded unless the size of the merging candidate list isone. Otherwise, if a PU is encoded with normal or regularinter-predicted mode, motion data for the PU, including motion vectordifference, reference picture index (indices), prediction directionflag, and an index (two indices) into the AMVP candidate lists for thePU, is encoded into bit stream. The entropy encoder 834 also encodes theCU and PU structure of each LCU.

The LCU processing includes an embedded decoder. As any compliantdecoder is expected to reconstruct an image from a compressed bitstream, the embedded decoder provides the same utility to the videoencoder. Knowledge of the reconstructed input allows the video encoderto transmit the appropriate residual energy to compose subsequentpictures. To determine the reconstructed input, i.e., reference data,the ordered quantized transform coefficients for a CU provided via thescan component 808 are returned to their original post-transformarrangement by an inverse scan component 810, the output of which isprovided to a dequantize component 812, which outputs a reconstructedversion of the transform result from the transform component 804.

The dequantized transform coefficients are provided to the inversetransform component 814, which outputs estimated residual informationwhich represents a reconstructed version of a residual CU. The inversetransform component 814 receives the transform block size used togenerate the transform coefficients and applies inverse transform(s) ofthe specified size to the transform coefficients to reconstruct theresidual values.

The reconstructed residual CU is provided to the combiner 838. Thecombiner 838 adds the delayed selected CU to the reconstructed residualCU to generate an unfiltered reconstructed CU, which becomes part ofreconstructed picture information. The reconstructed picture informationis provided via a buffer 828 to the intra-prediction component 824 andto an in-loop filter component 816. The in-loop filter component 816applies various filters to the reconstructed picture information toimprove the reference picture used for encoding/decoding of subsequentpictures. The in-loop filter component 816 may, for example, adaptivelyapply low-pass filters to block boundaries according to the boundarystrength to alleviate blocking artifacts causes by the block-based videocoding. The filtered reference data is provided to storage component818.

FIG. 9 shows a block diagram of an example video decoder. The videodecoder operates to reverse the encoding operations, i.e., entropycoding, quantization, transformation, and prediction, performed by thevideo encoder of FIG. 8 to regenerate the pictures of the original videosequence. In view of the above description of a video encoder, one ofordinary skill in the art will understand the functionality ofcomponents of the video decoder without detailed explanation.

The entropy decoding component 900 receives an entropy encoded(compressed) video bit stream and reverses the entropy coding to recoverthe encoded PUs and header information such as the prediction modes andthe encoded CU and PU structures of the LCUs, skip flags, merge flags,merge indices, etc. For each inter-predicted PU, unless skip mode isindicated, the entropy decoding component 900 decodes a merge flag fromthe bit stream. If the merge flag indicates that merge mode was notselected for the PU, the entropy decoding component 900 constructs anAMVP candidate list(s) for the PU and decodes a reference picture index(or indices), MVDs (motion vector difference), and an index (or indices)into the AMVP candidate list(s) for the PU. Construction of an AMVPcandidate list is previously described in reference to the encoder ofFIG. 8. The entropy decoder 900 then reconstructs the motion vector(s)according to the indicated AMVP candidate(s) and the decoded MVDs andprovides the motion vector(s) and reference picture index (or indices)to the motion compensation component 910.

If the merge flag indicates that merge mode was used for the PU in theencoder, the entropy decoding component 900 constructs a mergingcandidate list for the PU and decodes an index into the mergingcandidate list from the bit stream, if the index is in the bit stream.If no index is present, the index is assumed to be zero. Construction ofthe merging candidate list is previously described in reference to theencoder of FIG. 8. The entropy decoder provides the motion vector(s) andreference picture index (or indices) from the indicated mergingcandidate in the merging candidate list to the motion compensationcomponent 910.

If skip mode is indicated, the entropy decoding component 900 constructsa merging candidate list for the CU and decodes an index into themerging candidate list from the bit stream, if the index is in the bitstream. If no index is present, the index is assumed to be zero.Construction of the merging candidate list for skip mode is previouslydescribed in reference to the encoder of FIG. 8. The entropy decoderprovides the motion vector(s) and reference picture index (or indices)from the indicated merging candidate in the merging candidate list forskip mode to the motion compensation component 910.

The inverse quantization component 902 de-quantizes the quantizedtransform coefficients of the residual CU. The inverse transformcomponent 904 transforms the frequency domain data from the inversequantization component 902 back to the residual CU. That is, the inversetransform component 904 applies an inverse unit transform, i.e., theinverse of the unit transform used for encoding, to the de-quantizedresidual coefficients to produce the residual CUs.

A residual CU supplies one input of the addition component 906. Theother input of the addition component 906 comes from the mode switch908. When an inter-prediction mode is signaled in the encoded videostream, the mode switch 908 selects predicted PUs from the motioncompensation component 910 and when an intra-prediction mode issignaled, the mode switch selects predicted PUs from theintra-prediction component 914.

The motion compensation component 910 receives reference data fromstorage 912 and applies the motion compensation computed by the encoderand transmitted in the encoded video bit stream to the reference data togenerate a predicted PU. That is, the motion compensation component 910uses the motion vector(s) from the entropy decoder 900 and the referencedata to generate a predicted PU.

The intra-prediction component 914 receives reference data frompreviously decoded PUs of a current picture from the picture storage andapplies the intra-prediction computed by the encoder as signaled by theintra-prediction mode transmitted in the encoded video bit stream to thereference data to generate a predicted PU.

The addition component 906 generates a decoded CU by adding thepredicted PUs selected by the mode switch 908 and the residual CU. Theoutput of the addition component 906 supplies the input of the in-loopfilter component 916. The in-loop filter component 916 performs thefiltering as the encoder. The output of the in-loop filter component 916is the decoded pictures of the video bit stream. Further, the output ofthe in-loop filter component 916 is stored in storage 912 to be used asreference data.

Methods for derivation of a temporal motion data (TMD) candidate are nowdescribed. These methods may be used as part of the construction of aninter-prediction candidate list, i.e., a merging candidate list or anAMVP list, in both an encoder and a decoder. The methods of FIGS. 11-15provide TMD candidate derivation in which the TMD positions areconstrained to be within the boundaries of the co-located LCU row. Thatis, the fetch area for temporal motion data can go beyond the rightboundary of the co-located LCU into the right neighboring LCU but maynot extend into the bottom neighboring or right bottom neighboring LCUin the next LCU row. As is illustrated in FIG. 10, if one of thesemethods is used, an 80×64 co-located motion data area 1000 needs to bepre-fetched for a 64×64 LCU 1002.

Referring now to the method of FIG. 11, the method uses three TMDpositions, a primary TMD position that is the bottom right TMD positionoutside the co-located PU, a secondary TMD position that is the upperleft central TMD position within the co-located PU, and an alternativeTMD position that is considered if the primary TMD position is outsidethe co-located LCU row. In some embodiments, the alternative TMDposition may be a bottom right position outside the co-located PU andwithin the co-located LCU row. FIG. 12 shows an example partitioning ofan LCU with this alternative TMD position. In some embodiments, thealternative TMD position may the bottom right position inside theco-located PU. FIG. 13 shows an example partitioning of an LCU with thisalternative TMD position.

The TMD positions are defined as follows. Let (xP, yP) be the luminancecoordinates of the upper-left corner of the current PU in the currentpicture, and (puWidth, puHeight) be the size of the current PU. Theupper left central TMD position is defined as the luminance sampleposition in the co-located picture with the coordinates (xP+puWidth/2−1,yP+puHeight/2−1). The bottom right TMD position is defined as theluminance sample position in the co-located picture with the coordinates(xP+puWidth, yP+puHeight). The alternative TMD position of FIG. 12,i.e., the bottom right position outside the co-located PU and within theco-located LCU row, is defined as the luminance sample position in theco-located picture with the coordinates (xP+puWidth, yP+puHeight−1). Thealternative TMD position of FIG. 13, i.e., the bottom right positioninside the co-located PU, is defined as the luminance sample position inthe co-located picture with the coordinates (xP+puWidth−1,yP+puHeight−1). Motion data for a TMD position is copied from the PU inthe co-located picture which contains the TMD position.

Referring again to FIG. 11, initially, the primary TMD position, i.e.,the bottom right TMD position, is determined 1100. If the position iswithin the current LCU row 1102, then the availability 1104 of thebottom right TMD motion data is determined. In the examples of FIG. 12and FIG. 13, the bottom right TMD position is within the current LCU rowfor PU0, PU1, PU2, PU3, PU5 and PU6. Motion data for a TMD position maynot be available, for example, if the PU containing the TMD position wasintra-predicted or is outside the current slice or picture. If themotion data for bottom right TMD position is available, the motion datais returned 1112 to be used as the TMD candidate in the inter-predictioncandidate list being constructed.

If the bottom right TMD position is outside the current LCU row 1102,then the alternative TMD position is determined 1106. In the examples ofFIG. 12 and FIG. 13, the bottom right TMD position is outside thecurrent LCU row for PU4, PU7, and PU8. In some embodiments, thealternative TMD position is the bottom right position outside theco-located PU and within the LCU row as shown in FIG. 12. In someembodiments, the alternative TMD position is the bottom right positioninside the co-located PU as shown in FIG. 13. The availability of motiondata for the alternative TMD position is then determined 1104. If themotion data is available, the motion data is returned 1112 to be used asthe TMD candidate in the inter-prediction candidate list beingconstructed.

If the motion data of the bottom right TMD position or the alternativeTMD position (if considered), is not available 1104, then the secondaryTMD position, i.e., the upper left central TMD position, is determined1108. The availability of motion data for the upper left central TMDcandidate is then determined 1110. If the motion data is available, themotion data is returned 1112 to be used as the TMD candidate in theinter-prediction candidate list being constructed. Otherwise, anindication 1114 that no TMD candidate is available is returned.

Referring now to the method of FIG. 14, the method uses two TMDpositions, a primary TMD position, i.e., the bottom right TMD positionoutside the co-located PU and a secondary TMD position, i.e., the bottomright central TMD position within the co-located PU. The bottom rightcentral TMD position is defined as follows. Let (xP, yP) be theluminance coordinates of the upper-left corner of the current PU in thecurrent picture, and (puWidth, puHeight) be the size of the current PU.The bottom right central TMD position is defined as the luminance sampleposition in the co-located picture with the coordinates (xP+puWidth/2,yP+puHeight/2). FIG. 15 shows an example partitioning of an LCU and thetwo TMD positions relative to the example partitioning.

Referring again to FIG. 14, initially, the primary TMD position, i.e.,the bottom right TMD position, is determined 1400. If the position iswithin the current LCU row 1402, then the availability 1404 of motiondata for the bottom right TMD position is determined. In the example ofFIG. 15, the bottom right TMD position is within the current LCU row forPU0, PU1, PU2, PU3, PU5 and PU6. If motion data for the bottom right TMDposition is available, the motion data is returned 1412 to be used asthe TMD candidate in the inter-prediction candidate list beingconstructed.

If the bottom right TMD position is outside the current LCU row 1402 orthe bottom right TMD position is not available 1404, then the secondaryTMD position, i.e., the bottom right central TMD position, is determined1408. In the example of FIG. 15, the bottom right TMD position isoutside the current LCU row for PU4, PU7, and PU8. The availability ofmotion data for the bottom right central TMD position is then determined1410. If motion data for the bottom right central TMD position isavailable, the motion data is returned 1412 to be used as the TMDcandidate in the inter-prediction candidate list being constructed.Otherwise, an indication 1414 that no TMD candidate is available isreturned.

The methods of FIGS. 17-24 provide TMD candidate derivation in which theTMD positions are constrained to be within the boundaries of theco-located LCU and may not extend into any neighboring LCUs. That is,the fetch area for temporal motion data cannot go beyond the boundariesof the co-located LCU. As is illustrated in FIG. 16, if one of thesemethods is used, a 64×64 co-located motion data area 1600 needs to bepre-fetched for a 64×64 LCU 1602.

Referring now to the method of FIG. 17, the method uses three TMDpositions, a primary TMD position, i.e., the bottom right TMD positionoutside the co-located PU, a secondary TMD position, i.e., the upperleft central TMD position within the co-located PU, and an alternativeTMD position that is considered if the bottom right TMD position isoutside the co-located LCU. FIG. 18 shows an example partitioning of anLCU and the three TMD positions relative to the example partitioning.The alternative TMD position is a bottom right position inside theco-located PU (and within the co-located LCU).

Referring again to FIG. 17, initially, the primary TMD position, i.e.,the bottom right TMD position, is determined 1700. If the position iswithin the current LCU 1702, then the availability 1704 of motion datafor the bottom right TMD position is determined. In the example of FIG.18, the bottom right TMD position is within the current LCU for PU0,PU1, PU2, and PU5. If motion data for the bottom right TMD position isavailable, the motion data is returned 1712 to be used as the TMDcandidate in the inter-prediction candidate list being constructed.

If the bottom right TMD position is outside the current LCU 1702, thenthe alternative TMD position is determined 1706. In the example of FIG.18, the bottom right TMD position is outside the current LCU for PU3,PU4, PU6, PU7, and PU8. The alternative TMD position is the bottom rightposition within the co-located PU as shown in FIG. 18. If motion datafor the alternative TMD position is available, the motion data isreturned 1712 to be used as the TMD candidate in the inter-predictioncandidate list being constructed.

If motion data for the bottom right TMD position or the alternative TMDposition (if considered), is not available 1704, then the secondary TMDposition, i.e., the upper left central TMD position, is determined 1708.The availability of motion data for the upper left central TMD positionis then determined 1710. If motion data for the upper left central TMDposition is available, the motion data is returned 1712 to be used asthe TMD candidate in the inter-prediction candidate list beingconstructed. Otherwise, an indication 1714 that no TMD candidate isavailable is returned.

Referring now to the method of FIG. 19, the method uses two TMDpositions, a primary TMD position, i.e., the bottom right TMD positionoutside the co-located PU, and an alternative TMD position within theco-located PU that is considered if the bottom right TMD position isoutside the co-located LCU. FIG. 20 shows an example partitioning of anLCU and the two TMD positions relative to the example partitioning.

Referring again to FIG. 19, initially, the primary TMD position, i.e.,the bottom right TMD position, is determined 1900. If the position iswithin the current LCU 1902, then the availability 1904 of motion datafor the bottom right TMD position is determined. In the example of FIG.20, the bottom right TMD position is within the current LCU for PU0,PU1, PU2, and PU5. If motion data for the bottom right TMD position isavailable, the motion data is returned 1908 to be used as the TMDcandidate in the inter-prediction candidate list being constructed.Otherwise, an indication 1910 that no TMD candidate is available isreturned.

If the bottom right TMD position is outside the current LCU 1902, thenthe alternative TMD position, i.e., the bottom right TMD position insidethe co-located PU, is determined 1906. In the example of FIG. 20, thebottom right TMD position is outside the current LCU for PU3, PU4, PU6,PU7, and PU8. The availability of motion data for the alternative TMDposition is then determined 1904. If motion data for the alternative TMDposition is available, the motion data is returned 1908 to be used asthe TMD candidate in the inter-prediction candidate list beingconstructed. Otherwise, an indication 1910 that no TMD candidate isavailable is returned.

Referring now to the method of FIG. 21, the method uses two TMDpositions, a primary TMD position, i.e., the bottom right TMD positionwithin the co-located PU, and a secondary TMD position, i.e., the upperleft central TMD position within the co-located PU. FIG. 22 shows anexample partitioning of an LCU and the two TMD positions relative to theexample partitioning.

Referring again to FIG. 21, initially, the primary TMD position, i.e.,the bottom right TMD position inside the co-located PU, is determined2100. The availability 2102 of motion data for the TMD position is thendetermined. If motion data for the TMD position is available, the motiondata is returned 2108 to be used as the TMD candidate in theinter-prediction candidate list being constructed.

If motion data for the bottom right TMD position (inside the co-locatedPU) is not available 2102, then the secondary TMD position, i.e., theupper left central TMD position, is determined 2104. The availability ofmotion data for the upper left central TMD position is then determined2106. If motion data for the upper left central TMD position isavailable, the motion data is returned 2108 to be used as the TMDcandidate in the inter-prediction candidate list being constructed.Otherwise, an indication 2110 that no TMD candidate is available isreturned.

Referring now to the method of FIG. 23, the method uses a single TMDposition, the bottom right TMD position within the co-located PU. FIG.24 shows an example partitioning of an LCU and the TMD position relativeto the example partitioning.

Referring again to FIG. 23, initially, the bottom right TMD positioninside the co-located PU is determined 2300. The availability 2302 ofmotion data for the TMD position is then determined. If motion data forthe TMD position is available, the motion data is returned 2106 to beused as the TMD candidate in the inter-prediction candidate list beingconstructed. Otherwise, an indication 2304 that no TMD candidate isavailable is returned.

Embodiments of the methods, encoders, and decoders described herein maybe implemented for virtually any type of digital system (e.g., a desktop computer, a laptop computer, a tablet computing device, a netbookcomputer, a handheld device such as a mobile (i.e., cellular) phone, apersonal digital assistant, a digital camera, a set top box, a digitalvideo recorder, etc.). FIG. 25 is a block diagram of a digital system2500 (e.g., a mobile cellular telephone) that may be configured to usetechniques described herein.

As shown in FIG. 25, the signal processing unit (SPU) 2502 includes adigital signal processing system (DSP) that includes embedded memory andsecurity features. The analog baseband unit 2504 receives a voice datastream from the handset microphone 2513 a and sends a voice data streamto the handset mono speaker 2513 b. The analog baseband unit 2504 alsoreceives a voice data stream from the microphone 2514 a or 2532 a andsends a voice data stream to the mono headset 2514 b or wireless headset2532 b. The analog baseband unit 2504 and the SPU 2502 may be separateICs. In many embodiments, the analog baseband unit 2504 does not embed aprogrammable processor core, but performs processing based onconfiguration of audio paths, filters, gains, etc being setup bysoftware running on the SPU 2502.

The display 2520 may display pictures and video sequences received froma local camera 2528, or from other sources such as the USB 2526 or thememory 2512. The SPU 2502 may also send a video sequence to the display2520 that is received from various sources such as the cellular networkvia the RF transceiver 2506 or the Bluetooth interface 2530. The SPU2502 may also send a video sequence to an external video display unitvia the encoder unit 2522 over a composite output terminal 2524. Theencoder unit 2522 may provide encoding according to PAL/SECAM/NTSC videostandards.

The SPU 2502 includes functionality to perform the computationaloperations required for video encoding and decoding. In one or moreembodiments, the SPU 2502 is configured to perform computationaloperations for applying one or more techniques for temporal motion datacandidate derivation during the encoding process as described herein.Software instructions implementing all or part of the techniques may bestored in the memory 2512 and executed by the SPU 2502, for example, aspart of encoding video sequences captured by the local camera 2528. TheSPU 2502 is also configured to perform computational operations forapplying one or more techniques for temporal motion data candidatederivation as described herein as part of decoding a received codedvideo sequence or decoding a coded video sequence stored in the memory2512. Software instructions implementing all or part of the techniquesmay be stored in the memory 2512 and executed by the SPU 2502.

Other Embodiments

While the invention has been described with respect to a limited numberof embodiments, those skilled in the art, having benefit of thisdisclosure, will appreciate that other embodiments can be devised whichdo not depart from the scope of the invention as disclosed herein.

For example, although some methods for TMD candidate derivation aredescribed herein as using a primary TMD position that is the bottomright TMD position outside the co-located PU, other embodiments may usea different primary TMD position that is outside the co-located PU. Inanother example, although some methods for TMD candidate derivation aredescribed herein as using a secondary TMD position that is an upper leftcentral or a bottom right central TMD position, other embodiments mayuse a different secondary TMD position inside the co-located PU.Similarly, alternative TMD positions outside the co-located PU butwithin the LCU row or alternative TMD positions with the co-located LCUother than those described herein may be used in other embodiments.

Embodiments of the methods, encoders, and decoders described herein maybe implemented in hardware, software, firmware, or any combinationthereof. If completely or partially implemented in software, thesoftware may be executed in one or more processors, such as amicroprocessor, application specific integrated circuit (ASIC), fieldprogrammable gate array (FPGA), or digital signal processor (DSP). Thesoftware instructions may be initially stored in a computer-readablemedium and loaded and executed in the processor. In some cases, thesoftware instructions may also be sold in a computer program product,which includes the computer-readable medium and packaging materials forthe computer-readable medium. In some cases, the software instructionsmay be distributed via removable computer readable media, via atransmission path from computer readable media on another digitalsystem, etc. Examples of computer-readable media include non-writablestorage media such as read-only memory devices, writable storage mediasuch as disks, flash memory, memory, or a combination thereof.

Although the method steps may be presented and described herein in asequential fashion, one or more of the steps shown and described may beperformed concurrently, may be combined, and/or may be performed in adifferent order than the order shown in the figures and/or describedherein. Accordingly, embodiments should not be considered limited to thespecific ordering of steps shown in the figures and/or described herein.

It is therefore contemplated that the appended claims will cover anysuch modifications of the embodiments as fall within the true scope ofthe invention.

What is claimed is:
 1. A method, comprising: receiving a compressed bitstream comprising a picture; dividing the picture into one or morenon-overlapping blocks, wherein a row of the non-overlapping blocksincludes a first non-overlapping block; dividing the firstnon-overlapping block into one or more coding units; dividing a codingunit in the one or more coding units into one or more prediction units;determining a primary temporal motion data (TMD) candidate positionrelative to a first coordinate of a prediction unit of the one or moreprediction units in the picture; determining whether the primary TMDcandidate position is outside the row of the non-overlapping blocks inthe picture; and when the primary TMD candidate position is outside theco-located row: determining a secondary TMD candidate position relativeto a second coordinate inside the prediction unit in the picture; andselecting motion data related to the secondary TMD candidate position asan inter-prediction TMD candidate.
 2. The method of claim 1, wherein aco-located prediction unit in a reference picture covers coordinates ofthe secondary TMD candidate position and the motion data is related tothe co-located prediction unit.
 3. The method of claim 1, wherein thesecond coordinate inside the prediction unit is located in a centerpoint of the prediction unit.
 4. The method of claim 1, wherein thesecondary TMD candidate position coordinates ((xP+puWidth)/2,(yP+puHeight)/2), xP is an upper-left horizontal coordinate of theprediction unit of the one or more prediction units; yP is an upper-leftvertical coordinate of the prediction unit of the one or more predictionunits; puWidth is a width of the prediction unit of the one or moreprediction units; and puHeight is a height of the prediction unit of theone or more prediction units.
 5. The method of claim 1, wherein thefirst coordinate of the prediction unit is a bottom-right coordinate ofthe prediction unit.
 6. The method of claim 1, wherein the primary TMDcandidate position has coordinates (xP+puWidth, yP+puHeight), xP is anupper-left horizontal coordinate of the prediction unit of the one ormore prediction units; yP is an upper-left vertical coordinate of theprediction unit of the one or more prediction units; puWidth is a widthof the prediction unit of the one or more prediction units; and puHeightis a height of the prediction unit of the one or more prediction units.7. A system, comprising: a receiver configured to receive a compressedbit stream comprising a picture; and a decoder coupled to the receiver,the decoder configured to: divide the picture into one or morenon-overlapping blocks, wherein a row of the non-overlapping blocksincludes a first non-overlapping block; divide the first non-overlappingblock into one or more coding units; divide a coding unit in the one ormore coding units into one or more prediction units; determine a primarytemporal motion data (TMD) candidate position relative to a firstcoordinate of a prediction unit of the one or more prediction units inthe picture; determine whether the primary TMD candidate position isoutside the row of the non-overlapping blocks in the picture; and whenthe primary TMD candidate position is outside the co-located row:determine a secondary TMD candidate position relative to a secondcoordinate inside the prediction unit in the picture; and select motiondata related to the secondary TMD position as an inter-prediction TMDcandidate.
 8. The system of claim 7, wherein a co-located predictionunit in a reference picture covers coordinates of the secondary TMDposition and the motion data is related to the co-located predictionunit.
 9. The system of claim 7, wherein the second coordinate inside theprediction unit is located in a center point of the prediction unit. 10.The system of claim 7, wherein the secondary TMD candidate position hascoordinates ((xP+puWidth)/2, (yP+puHeight)/2), xP is an upper-lefthorizontal coordinate of the prediction unit of the one or moreprediction units; yP is an upper-left vertical coordinate of theprediction unit of the one or more prediction units; puWidth is a widthof the prediction unit of the one or more prediction units; and puHeightis a height of the prediction unit of the one or more prediction units.11. The system of claim 7, wherein first coordinate of the predictionunit is a bottom-right coordinate of the prediction unit.
 12. The systemof claim 7, wherein the primary TMD candidate position has coordinates(xP+puWidth, yP+puHeight), xP is an upper-left horizontal coordinate ofthe prediction unit of the one or more prediction units; yP is anupper-left vertical coordinate of the prediction unit of the one or moreprediction units; puWidth is a width of the prediction unit of the oneor more prediction units; and puHeight is a height of the predictionunit of the one or more prediction units.
 13. A system, comprising: areceiver configured to receive a compressed bit stream comprising apicture; and a decoder coupled to the receiver, the decoder configuredto: divide the picture into one or more non-overlapping blocks, whereina row of the non-overlapping blocks includes a first non-overlappingblock; divide the first non-overlapping block into one or more codingunits; divide a coding unit in the one or more coding units into one ormore prediction units; determine a primary temporal motion data (TMD)candidate position relative to a first coordinate of a prediction unitof the one or more prediction units in the picture; determine whetherthe primary TMD candidate position is outside the row of thenon-overlapping blocks in the picture; when the primary TMD candidateposition is outside the co-located row: determine a secondary TMDcandidate position relative to a second coordinate inside the predictionunit in the picture; and select motion data related to the secondary TMDposition as an inter-prediction TMD candidate; and reconstruct thepicture using the selected motion data; and a display coupled to thedecoder, the display configured to display the reconstructed picture.14. The system of claim 13, wherein a co-located prediction unit in areference picture covers coordinates of the secondary TMD position andthe motion data is related to the co-located prediction unit.
 15. Thesystem of claim 13, wherein the second coordinate inside the predictionunit is located in a center point of the prediction unit.
 16. The systemof claim 13, wherein the secondary TMD candidate position hascoordinates ((xP+puWidth)/2, (yP+puHeight)/2), xP is an upper-lefthorizontal coordinate of the prediction unit of the one or moreprediction units; yP is an upper-left vertical coordinate of theprediction unit of the one or more prediction units; puWidth is a widthof the prediction unit of the one or more prediction units; and puHeightis a height of the prediction unit of the one or more prediction units.17. The system of claim 13, wherein the first coordinate of theprediction unit is a bottom-right coordinate of the prediction unit. 18.The system of claim 13, wherein the primary TMD candidate position hascoordinates (xP+puWidth, yP+puHeight), xP is an upper-left horizontalcoordinate of the prediction unit of the one or more prediction units;yP is an upper-left vertical coordinate of the prediction unit of theone or more prediction units; puWidth is a width of the prediction unitof the one or more prediction units; and puHeight is a height of theprediction unit of the one or more prediction units.